[DRAFT] Analysis of Software and Data Carpentry’s Pre- and Post-Workshop Surveys

Authors: The Carpentries Assessment Team

Published: July 2018

Overview

Since XXXX, Software and Data Carpentry have collected information on learner demographics, perception of tools and confidence in working with data. As we continue in our goal to streamline processes as The Carpentries, the Assessment Team completed an analysis of the pre- and post-workshop surveys for both Software and Data Carpentry. The goal of this analysis is to understand the impact our workshops are having on learners, and how we can improve our surveys and assessment infrastructure. This report covers the workshops from NA to NA.

As an overview, 1495 learners have responded to Data Carpentry’s surveys, while 14185 have responded to Software Carpentry’s.

This report includes the following:

• Motivation for Attending Carpentries Workshops
• Workshop Type and Perception of Workshop Environment/Experience
• Effect of Workshops on Learner’s Self-Reported Perspectives, Skills, and Confidence
• Ability to Perform Computing Tasks
• Demographics
• Summary

Motivation for Attending Carpentries Workshops

Learners attend Carpentries workshops for many reasons. Data Carpentry’s workshops are domain specific and focus on the fundamental data skills needed to conduct research. Data Carpentry’s Ecology and Social Sciences curricula begin with a lesson on data organization and includes data cleaning with OpenRefine. From there, learners spend time learning a base programming language, either Python or R.

Data Carpentry’s Genomics curriculum also includes programming, however the focus of this curriculum is best practices for organization of bioinformatics projects and data, use of command line utilities and tools to analyze sequence quality and perform variant calling, and connecting to and using cloud computing.

Why learners attend Data Carpentry workshops	n	%
To learn skills that I can apply to my work in the future	960	63.7
To learn skills that will help me get a job	445	29.5
As a requirement for my program/current position	103	6.8

XX% of Data Carpentry learners attend workshops to learn skills they can apply to their work in the future.

Software Carpentry workshops teach automation with the Unix shell, a tool that allows users to run commands interactively or by scripting. In Software Carpentry workshops, version control of source code with Git and GitHub are also taught for learners to learn facilitating contribution and collaboration on online repositories. Programming in R or Python is also taught. Software Carpentry’s curriculum teaches basic lab skills for scientific computing. Compared to Data Carpentry’s learners, Software Carpentry’s tend to have more experience witht the tools covered in the workshops, and learners come to learn new and/or additional topics (XX%). It’s also interesting to note that XX% of Software Carpentry respondents are first-time learners.

Why learners attend Software Carpentry Workshops	n	%
To cover new/additional topics	458	74.7
To network	78	12.7
To become a Software Carpentry helper/instructor	39	6.4
To help host/run a workshop	38	6.2

Software Carpentry 1st Time Learners	n	%
Yes	11525	94
No	697	6

The majority (XX%) of Data Carpentry’s respondents are either unsatisfied or feel neutral about being satisfied with their current data management practices. By data management practices, we include behaviors such as keeping your raw data raw, reusing code, and using databases, queries, and scripts to manage large datasets.

Data Carpentry Learners satisfaction with current data management practices	n	%
Very unsatisfied	100	6.7
Unsatisfied	408	27.3
Neutral	405	27.1
Satisfied	173	11.6
Very satisfied	22	1.5
Not sure	75	5.0
Not applicable	60	4.0
Didn’t answer	252	16.9

In terms of current programming usage, XX% of learners either never use programming, or use programming less than once per year, but no more than several times per year. Only XX% program on a daily basis. This is no surprise, as Data Carpentry workshops tend to attract novices.

Data Carpentry Learners current programming usage	n	%
Never	314	31.9
Less than once per year	151	15.4
Several times per year	168	17.1
Monthly	101	10.3
Weekly	131	13.3
Daily	118	12.0

Data Carpentry workshops are mainly populated by “word-of-mouth”. XX% of respondents learned about the workshop from either a friend, colleague, or their advisor or supervisor.

How respondents find out about Data Carpentry workshops	n	%
My friend/colleague told me about it	324	47.0
My advisor/supervisor told me about it	244	35.4
Read about it in a newsletter or university web site	85	12.3
Other web site	20	2.9
Twitter or other social media	17	2.5

On the contrary, the majority of Software Carpentry workshop respondents (XX%) find out about workshops through instituion mailing lists or flyers.

How respondents find out about Software Carpentry workshops	n	%
Institution mailing list or flyer	5152	75.3
Conference/meeting/seminar	641	9.4
Funding organization or program officer	357	5.2
Our website	357	5.2
Social Media (Twitter, Facebook, etc.)	294	4.3
Journal or publication	37	0.5

In summary, both Data and Software Carpentry workshop respondents attend workshops to learn about or improve upon their current data management and analysis skills.

Workshop Type and Perception of Workshop Environment/Experience

As previously mentioned, Data Carpentry workshops are domain specific, and curricula include Ecology, Genomics, Geospatial, Social Sciences, and Reproducible Research. XX% of respondents learned R in their workshop, while XX% learned Python.

Data Carpentry: Language Covered in Workshops	n	%
R	808	64.7
Python	188	15.1
I don’t know/I don’t remember	218	17.5
Neither	35	2.8

Workshop Environment

The Carpentries is committed to making participation in our workshops a harassment-free experience for everyone, regardless of who you are, where you come from, or your experience with the tools we teach. We establish norms for interaction by having, discussing, and enforcing a Code of Conduct such that our workshops provide open and inclusive learning environments. XX% of Data Carpentry respondents either agree or strongly agree that they felt comfortable learning in their workshop environment, and XX% of Software Carpentry’s respondents agreed or strongly agreed the workshop atmosphere was welcoming.

Data Carpentry respondents were asked to rate their level of agreement with several statements regarding their instructor’s knowledge, instructional method, and enthusiasm. Their responses are in the figure below, and axis labels correspond to the statements as follows:

• Instructors Knowledge: The instructors were knowledgeable about the material being taught.
• Instructors Interacting: I felt comfortable interacting with the instructors.
• Instructors Enthusiastic: The instructors were enthusiastic about the workshop.
• Instructors Clear Answers: I was able to get clear answers to my questions from the instructors.

The largest impact we see is that XX% of respondents said they felt comfortable interacting with the instructors. We know that our instructors are the reason why our workshops are so well received. It’s also great to see that XX% and XX% of respondents felt our instructors were knowledgeable about the material being taught, and were enthusiastic about the workshop, respectively. We would like to explore what training would help our instructors so that the percentage of respondents who felt they were able to get clear answers to their questions from the instructors would increase.

Software Carpentry respondents were asked to rate how they felt instructors and helpers worked as a team based on the following criteria:

• Considerate: Instructors/Helpers were considerate.
• Enthusiastic: Instructors/Helpers were enthusiastic.
• Clear Answers: Instructors/Helpers gave clear answers to your questions.
• Communicators: Instructors/Helpers were good communicators.

The two Likert plots below provide an analysis of respondent’s answers for both instructors and helpers.

From the figures above, we see that Software Carpentry instructors and helpers are considerate, enthusiastic, give clear answers to questions, and are good communicators. As a whole, our instructors work as a team and are successful in creating a warm and welcoming workshop environment.

One of the goals for Data Carpentry’s lessons is that learners are able to immediately apply what they learned at the workshop. The figure below shows that XX% either agree or strongly agree that they’re able to immediately apply what they learned.

As the majority of Software Carpentry learners attend workshops to learn new skills it is great to see that XX% of learners either learned mostly or all new information during the workshop.

Workshop Experience

We want to be proactive in ensuring learners have access to whatever they need to participate in a workshop. Both Data and Software Carpentry learners are asked to inform workshop organizers if there is anything they need that would make their workshop experience better. Data Carpentry’s respondents were asked if they had accessibility issues, and XX% reported they did. After reading the open-ended responses, we see that the issues were related to not being able to hear and/or see in the back of the room.

Data Carpentry Respondents Having Accessibility Issues	n	%
No	654	89.2
Yes	79	10.8

We use the Net Promoter Score to measure learners’ likelihood of recommending workshops to a friend or colleague. The scoring for this question is on a 0 to 100 scale. Respondents scoring from 0 to 64 are labeled Detractors, and are believed to be less likely to recommend a workshop. Those who respond with a score of 85 to 100 are called Promoters, and are considered likely to recommend a workshop. Respondents between 65 and 84 are labeled Passives, and their behavior falls in the middle of Promoters and Detractors.

Data Carpentry Promoter Score	n	%
Detractor	32	4.395604
Passive	131	17.994505
Promoter	565	77.609890

78% of Data Carpentry respondants are promoters (i.e. would recommend a workshop).

Software Carpentry Promoter Score	n	%
Detractor	549	16.69708
Passive	634	19.28224
Promoter	2105	64.02068

For Software Carpentry respondents, 64% are promoters.

In summary, Data and Software Carpentry workshops provide a warm and welcoming environment whether learners are brand new to programming or have some experience. Attendees are recommending workshops to their friends and colleagues, and we know that our instructors and helpers are the major reason why.

Effect of Workshops on Learners Self-Reported Perspectives: Skills & Confidence

In our workshops, we recommend that learners use their own machines. It is important for learners to leave the workshop with their own machine set up to do real work. Our instructors teach on three major platforms: UNIX/Linux, Mac OS X, and Windows. We see a very close representation of Windows (XX%) and Apple/Mac OS (XX%) users in our Data Carpentry workshops, and even a few UNIX/Linux users (XX%).

OS Respondents Use in Data Carpentry Workshops	n	%
Windows	661	53.3
Apple/Mac OS	512	41.3
UNIX/Linux	50	4.0
Not sure	17	1.4

Learners were asked to rate their level of agreement with the following statements related to Data Carpentry’s workshop goals and learning objectives. The figure below provides a visual representation of their responses, comparing them before the workshop and after the workshop. Axis labels and the corresponding question are as follows:

• Write Program: I can write a small program/script/macro to solve a problem in my own work.
• Search Online: I know how to search for answers to my technical questions online.
• Raw Data: Having access to the original, raw data is important to be able to repeat an analysis.
• Programming Efficient: Using a programming language (like R or Python) can make me more efficient at working with data.
• Programming Confident: I am confident in my ability to make use of programming software to work with data.
• Overcome Problem: While working on a programming project, if I get stuck, I can find ways of overcoming the problem.
• Analyses Easier: Using a programming language (like R or Python) can make my analyses easier to reproduce.

As the scoring for the above factors is ordinal from strongly disagree (1) to strongly agree (5), we show the mode (most frequent responses) for respondents’ before the workshop, and after the workshop. If there is only one point on the plot it is because the mode for pre- and post-workshop responses was the same. The comparison above is paired, meaning, we are comparing those who provided us with a unique identifier and who completed both the pre- and post-workshop survey. This figure includes XX respondses.

In the figures below we show another representation of the pre- and post-comparison of respondents skills and perspectives. The figures below include the data for all learners, not only those who provided a unique identifier and took both the pre- and post-workshop surveys. What we see is a shift in the distribution of for each factor, meaning, respondents’ self-reported confidence and ability shifted in a positive directions.

Another representation of the positive shift in distribution is provided below.

The neutral centered graphs below provide an even clearer picture of the shift in respondents self-reported confidence and skills.

It is interesting to see the shift in neutrality between the pre-workshop scores and post-workshops scores, especially for Programming Efficient. There was a higher percentage of learners beginning the workshop who felt programming with R or Python can make them more efficient at working with data. Contrarily, confidence in using programming to work with data increased from XX% to xx%.

Software Carpentry Respondets were asked to tell us about their experience with these topics before the workshop:

• R
• Unix Shell
• SQL
• Python
• Version Control with Git

From the figure we see that XX% and less had extensive knowledge of the topics covered in their workshop.

The following is a comparison of Software Carpentry Respondents’ knowledge about the tools before compared to after the workshop. We see clearly that after the workshop, respondents’ knowledge of Git, Python, R, and the Unix Shell increased a great deal.

Respondent Ability to Perform Computing Tasks

Motivation is important, but being confident in your ability to complete specific computing tasks is an equally important goal of Software Carpentry. The grid below shows respondents’ self-reported ability to complete tasks including:

• Using pipes to connect shell commands
• Writing a ‘for loop’ to automate tasks
  
• Initializing a repository with git
• Writing a function
• Importing a library or package in R or Python
• Writing a unit test in Python or R
• Writing an SQL query

It also provides their self-reported level of confidence in being able to complete the tasks above after completing the workshop.

These figures tell us that, before the workshop, between XX% and XX% of the respondents did not feel they could initialize a repository in Git, write a ‘for loop’ to automate tasks, use pipes to connect shell commands, write a SQL query, and/or write a unit test in R or Python. XX% of learners felt their confidence increased greatly with respect to importing a library or package in R or Python. We consider this significant as it is one of the fundamental skills that allows learners to be successful in the other areas mentioned above.

In summary, respondents experienced increased confidence in their ability to perform specific computing tasks and solve problems, or at least search for answers to problems, as a result of participating in Software and Data Carpentry workshops.

Demographics

The Carpentries is a global community that has recognized the importance of bringing people to data through high-impact trainings. Thought the majority of Data Carpentry respondents attended a workshop in the United States of America (XX%), we see an increase in workshops in places like Ethiopia (XX%), Switzerland (XX%), and India (XX%).

In Software Carpentry’s pre-workshop survey, respondents are asked whether or not their workshop takes place in the United States. XX% of respondents attended a U.S. workshop.

Software Carpentry Workshops in US	n	%
Yes	6489	45.7
No	4650	32.8
Didn’t answer	3046	21.5

As previously mentioned, Data Carpentry’s curricula is domain specific to Ecology, Genomics, Geospatial, and the Social Sciences. We see this in the distribution of respondents by discipline. XX% are in the Life Science, while XX%, XX%, and XX% are in Agricultural or Environmental Sciences, Bioinformatics/Genomics, and Biomedical/Health Sciences, respectively.

Data Carpentry’s Respondents by Discipline	n	%
Life Sciences	444	22.0
Agricultural or Environmental Sciences	307	15.2
Bioinformatics/Genomics	292	14.5
Biomedical/Health Sciences	288	14.3
Social Sciences	122	6.0
Mathematics or Statistics	101	5.0
Earth Sciences	96	4.8
Engineering	91	4.5
Computer Science	88	4.4
Business/Economics	57	2.8
Humanities	53	2.6
Physical Sciences	53	2.6
Library Sciences	28	1.4

Software Carpentry’s respondent base also has a majority Life Sciences base (XX%), however we also see representation from those working in Psychology, High Performance Computing, and Chemistry.

Software Carpentry’s Respondents by Discipline	n	%
Life Science - Organismal/systems (ecology, botany, zoology, microbiology, neuroscience)	2698	21.0
Life Sciences (Genetics, genomics, bioinformatics )	2683	20.9
Mathematics/statistics	945	7.4
Physics	803	6.2
Planetary sciences (geology, climatology, oceanography, etc.)	787	6.1
Civil, mechanical, chemical, or nuclear engineering	696	5.4
Medicine and/or Pharmacy	688	5.4
Social sciences	591	4.6
Chemistry	578	4.5
Economics/business	483	3.8
Psychology	418	3.3
Library and information science	375	2.9
High performance computing	365	2.8
Humanities	319	2.5
Education	264	2.1
Space sciences	163	1.3

As many of The Carpentries’ workshops are hosted on college campuses and other research-based communities, there is no surprise that the majority of respondents are Graduate Students (XX% - DC, XX% - SWC), Research Staff (XX% - DC, XX% - SWC), and Postdoctoral Researchers (XX% - DC, XX% - SWC).

Data Carpentry’s Respondents by Position	n	%
Graduate Student	592	45.9
Research Staff	200	15.5
Postdoctoral Researcher	183	14.2
Faculty	101	7.8
Government Employee	80	6.2
Industry Employee	49	3.8
Undergraduate Student	48	3.7
Management/Administrator	20	1.5
Retired/Not Employed	18	1.4

Gender and racial/ethnic identity information is collected for U.S. participants, as we are keen to increase the number of diverse instructors and learners we serve. Understanding our demographic makeup helps us to strategize about what communities to reach and what programs to offer.

Currently, both Data (XX%) and Software (XX%) Carpentry see strong representation from Women in the United States. Where we hope to improve is in reaching the non-White audience, as less than XX% of our respondents are from communities historically underrepresented in the science, technology, engineering, and mathematics (STEM) fields.

Data Carpentry’s U.S. Respondents’ Gender Identity	n	%
Female	322	58
Male	223	40
Transgender female	2	0
Prefer not to answer	8	1

Data Carpentry’s U.S. Respondents Racial/Ethnic Identity	n	%
White	316	54.0
Asian	152	26.0
Hispanic or Latino(a)	57	9.7
I prefer not to say.	28	4.8
Black or African American	25	4.3
American Indian or Alaska Native	4	0.7
Native Hawaiian or Other Pacific Islander	3	0.5

Software Carpentry’s U.S. Respondents’ Gender Identity	n	%
Female	322	58
Male	223	40
Transgender female	2	0
Prefer not to answer	8	1

Software Carpentry’s U.S. Respondents’ Racial/Ethnic Identity	n	%
American Indian or Alaskan Native	16	0
Asian / Pacific Islander	751	23
Black or African American	144	4
Hispanic or Latino	204	6
Native Hawaiian or Other Pacific Islander	3	0
White / Caucasian	1916	58
Prefer not to say	197	6
Multiple ethnicity / Other (please specify)	95	3

Summary